Skip to content

Respect GitHub rate-limit headers in responses #233

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 12 commits into
base: master
Choose a base branch
from
Open

Respect GitHub rate-limit headers in responses #233

wants to merge 12 commits into from

Conversation

ericphanson
Copy link
Collaborator

closes #231

written with claude code, though with a lot of hand-holding.

Example of hitting the rate limit in real life:

julia> commits(Repo("JuliaRegistries/RegistryCI.jl"));
┌ Info: GitHub API primary rate limit hit, retrying in 3070.6s
│   method = "GET"
│   url = "https://api.github.com/repositories/208115424/commits?page=27"
│   status = 403
│   limit = "60"
│   remaining = "0"
│   used = "60"
│   reset = "1759013549"
└   resource = "core"
^CERROR: InterruptException:
Stacktrace:
  [1] try_yieldto(undo::typeof(Base.ensure_rescheduled))
    @ Base ./task.jl:958
  [2] wait()
    @ Base ./task.jl:1022
  [3] wait(c::Base.GenericCondition{Base.Threads.SpinLock}; first::Bool)
    @ Base ./condition.jl:130
  [4] wait
    @ ./condition.jl:125 [inlined]
  [5] _trywait(t::Timer)
    @ Base ./asyncevent.jl:145
  [6] wait
    @ ./asyncevent.jl:162 [inlined]
  [7] sleep(sec::Float64)
    @ Base ./asyncevent.jl:272
  [8] with_retries(f::GitHub.var"#21#23"{…}; method::String, url::SubString{…}, max_retries::Int64, verbose::Bool)
    @ GitHub ~/GitHub.jl/src/utils/requests.jl:253
  [9] with_retries
    @ ~/GitHub.jl/src/utils/requests.jl:214 [inlined]
 [10] make_request_with_retries
    @ ~/GitHub.jl/src/utils/requests.jl:331 [inlined]
 [11] github_paged_get(api::GitHub.GitHubWebAPI, endpoint::String; page_limit::Float64, start_page::String, handle_error::Bool, auth::GitHub.AnonymousAuth, headers::Dict{…}, params::Dict{…}, max_retries::Int64, options::@Kwargs{})
    @ GitHub ~/GitHub.jl/src/utils/requests.jl:350
 [12] github_paged_get
    @ ~/GitHub.jl/src/utils/requests.jl:322 [inlined]
 [13] gh_get_paged_json(api::GitHub.GitHubWebAPI, endpoint::String; options::@Kwargs{})
    @ GitHub ~/GitHub.jl/src/utils/requests.jl:368
 [14] gh_get_paged_json
    @ ~/GitHub.jl/src/utils/requests.jl:367 [inlined]
 [15] commits(api::GitHub.GitHubWebAPI, repo::Repo; options::@Kwargs{})
    @ GitHub ~/GitHub.jl/src/repositories/commits.jl:32
 [16] commits
    @ ~/GitHub.jl/src/repositories/commits.jl:31 [inlined]
 [17] commits(repo::Repo)
    @ GitHub ./none:0
 [18] top-level scope
    @ REPL[5]:1
Some type information was truncated. Use `show(err)` to see complete types.

@ericphanson
Copy link
Collaborator Author

ericphanson commented Sep 28, 2025 •
edited
Loading

I think I should add a limit that if we want to sleep more than say 15 mins, we error instead with a clear error message. Since in many cases it would be unexpected that your task will just sleep for 12 hours or something like that. Super long waits can happen when you use up your quota and it wants to wait for a refresh. This value could be configurable with some “reasonable” default.

edit: done in 416b21e

This was referenced Sep 28, 2025
Draft
Open
@DilumAluthge DilumAluthge reopened this Oct 6, 2025
@DilumAluthge
Copy link
Member

@ericphanson Can you also try this functionality out from a non-fork PR, so we can see if there's any behavior difference between a user's PAT and the GitHub Actions GITHUB_TOKEN? I'm hoping there's no difference, but it'd be good to check.

@DilumAluthge
Copy link
Member

It looks like CI is failing because we are rate-limited when calling GitHub.authenticate(token).

GitHub.authenticate() calls gh_get():

GitHub.jl/src/utils/auth.jl

Lines 65 to 69 in 942e814

@api_default function authenticate(api::GitHubAPI, token::AbstractString; options...)
auth = OAuth2(token)
gh_get(api, "/"; auth = auth, options...)
return auth
end

Which in turn should call github_request():

GitHub.jl/src/utils/requests.jl

Line 78 in 942e814

gh_get(api::GitHubAPI, endpoint = ""; options...) = github_request(api, HTTP.get, endpoint; options...)

@ericphanson Would it be possible to add your rate-limiting logic to github_request() also?

DilumAluthge
DilumAluthge reviewed Oct 6, 2025
View reviewed changes
src/utils/requests.jl Outdated
delay_seconds = safe_tryparse(Float64, retry_after)
if delay_seconds !== nothing
delay_seconds = parse(Float64, retry_after)
verbose && @info "$msg, retrying in $(round(delay_seconds, digits=1))s" method=method status=status limit=limit remaining=remaining used=used reset=reset_time resource=resource retry_after=retry_after
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you Dates.canonicalize() the delay_seconds, to make it easier to read?

DilumAluthge
DilumAluthge reviewed Oct 6, 2025
View reviewed changes
@ericphanson ericphanson mentioned this pull request Oct 6, 2025
Open
@ericphanson
Copy link
Collaborator Author

ericphanson commented Oct 6, 2025 •
edited
Loading

@ericphanson Can you also try this functionality out from a non-fork PR, so we can see if there's any behavior difference between a user's PAT and the GitHub Actions GITHUB_TOKEN? I'm hoping there's no difference, but it'd be good to check.

#249

@ericphanson Would it be possible to add your rate-limiting logic to github_request() also?

It is there! Every API request in this package should be covered. That's the with_retries block in github_request. Where do you see the failure? Send a link if you can

@codecov Codecov
Copy link

codecov bot commented Oct 6, 2025 •
edited
Loading

Codecov Report

❌ Patch coverage is 90.10989% with 9 lines in your changes missing coverage. Please review.
✅ Project coverage is 58.85%. Comparing base (942e814) to head (a7e1452).

Files with missing lines Patch % Lines
src/utils/requests.jl 90.10% 9 Missing ⚠️
Additional details and impacted files 
@@            Coverage Diff             @@
##           master     #233      +/-   ##
==========================================
+ Coverage   55.19%   58.85%   +3.66%     
==========================================
  Files          37       37              
  Lines         904      982      +78     
==========================================
+ Hits          499      578      +79     
+ Misses        405      404       -1

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@DilumAluthge DilumAluthge Awaiting requested review from DilumAluthge

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Respect GitHub's rate limit headers
2 participants
@ericphanson @DilumAluthge